RTSI: An Index Structure for Multi-Modal Real-Time Search on Live Audio Streaming Services

Audio streaming services (e.g., Mixlr, Ximalaya, Lizhi and Facebook Live Audio) have become increasingly popular due to the wide use of smart phones. More and more people are enjoying live audio broadcasting while they are doing various kinds of activities. On the other hand, the data volume of live audio streams is also ever increasing. Searching and indexing these audio streams is still an important and open problem, with the following challenges: (i) queries on the large number of audio streams need to be answered in real-time; (ii) a live audio stream is inserted into the index continuously to enable live audio streams to appear in query results, and the number of insertions is large which often becomes a performance issue. Existing studies on audio search either oversimplify the problem or simply ignore searching live audio streams. Moreover, existing studies do not explore the multi-modal property of audio streams. In this application paper, we propose a multi-modal and unified log structured merge-tree (a.k.a. LSM-tree which consists of multiple inverted indices) based index to support intensive insertions and real-time search on live audio stream applications. Our index natively supports two major types of indexing techniques in audio search: text based indexing and sound based indexing. The key technical challenges we need to address are that (i) a live audio stream may appear in multiple inverted indices due to the streaming nature, (ii) relevance, popularity and freshness of each audio stream need to be maintained in a way that allows fast accesses, and a query often matches to a large number of audio streams since audio streams usually contain many unique terms, and (iii) massive insertions are happening alongside with queries. To address the above challenges, we propose an index (called RTSI) which avoids traversing multiple inverted indices to compute the score of an audio stream. In RTSI, we propose various techniques to address the technical challenges. First, we use one inverted list which contains the sorted score of popularity, freshness and relevance, such that we can compute the top-k query results efficiently. Second, we devise an upper bound for the unchecked audio streams, such that the query answering process can be terminated early. Third, we create mirrors for the indices that need to be merged, such that queries can be answered in real-time when the indices are merging. We conduct extensive experiments on audio streams obtained from Ximalaya. The experimental results show that RTSI can answer a large number of queries in a real-time manner while concurrently handling massive insertions.